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INTRODUCTION 


Understanding  normal  biology  and  its  pathological  deviations  requires 
comprehending  the  form  in  which  the  different  tissue  components  work  together  in  their 
native  tissue  context.  In  particular,  it  requires  studying  cell-to-cell  and  cell-to- 
environment  interactions,  which  tend  to  be  highly  variable  between  different  tissue  parts. 
This  information  can  only  be  acquired  by  studying  tissue  in  a  state  as  similar  to  its  native 
context  as  possible. 

Contextual  information,  which  is  important  for  understanding  normal  tissue 
behavior,  becomes  essential  when  studying  Breast  Cancer,  since  heterogeneity  and 
three-dimensionality  are  at  the  very  base  of  cancer  initiation  and  clonal  progression. 
However,  many  analytical  procedures  in  biology  start  by  taking  the  element  under  study 
(RNA,  DNA,  etc)  outside  its  native  context.  By  doing  so,  an  accurate  measure  of  that 
element  is  achieved,  but  priceless  information  on  how  that  element  is  related  to  its 
environment  is  lost.  Other  methods  based  on  immunohistochemical  staining  of  thin 
tissue  sections  for  visual  observation  at  the  conventional  microscope  preserve  only 
partial  contextual  information,  since  they  provide  a  flat  -two  dimensional-  view  of  a  three- 
dimensional  object. 

Our  goal  is  to  develop  a  three-dimensional  computer-based  quantitative 
microscopy  platform  to  be  applied  to  simultaneous  morphological  and  molecular  studies 
of  both  normal  and  neoplastic  mammary  tissue.  Using  our  system,  we  will  be  able  to 
reconstruct  relevant  microscopic  tissue  structures  (e.g.  mammary  ducts,  lobules,  lymph 
nodes,  tumor  masses,  etc.)  from  consecutive  thin  tissue  sections.  Then,  using  the  3D 
virtual  reconstruction,  it  will  be  possible  to  perform  quantitative  morphological 
measurements  combined  with  directed  -structure  defined-  high-resolution  analysis  of 
molecular  events. 

To  show  the  use  of  our  system,  we  will  study  the  combined  role  of  Estrogen  and 
Progesterone  Receptors  (ER  &  PR)  in  mammary  gland  development.  We  will  quantify 
cell-to-cell  the  levels  of  ER  and  PR  in  different  parts  of  the  mammary  gland,  at  different 
time  points  of  the  development  of  the  gland. 
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BODY 


Our  accomplishments  during  the  first  year  (08/01/00-07/31/01)  of  the  project  will  be 
described  following  the  Tasks  enumerated  in  the  approved  proposal.  The  tasks  are  listed 
below,  and  the  sub-tasks  corresponding  to  the  first  funding  year  have  been  underlined. 
The  text  has  references  to  the  numbered  tasks  (e.g.  see  Task  1. 1)  where  the  fulfillment 
of  the  tasks  is  explained.  Before  starting  describing  them,  it  has  to  be  noted  that  most  of 
the  goals  have  been  amply  fulfilled,  including  some  that  were  scheduled  for  the  second 
year,  even  though  the  effective  start  date  of  the  work  was  10/01/01,  i.e.  two  months  after 
the  administrative  start  date.  This  has  been  possible  in  part  thanks  to  the  liberality  of  the 
contract  officers,  which  allowed  us  to  join  the  budgetary  efforts  of  this  and  our  other 
grant,  “Mechanisms  of  Intraductal  Tumor  Spread"  (DAMD1 7-00-1 -0227),  which  goals  for 
the  first  year  partially  overlapped  with  the  goals  of  this  grant. 

Task  1.  (Months  1-12)  Modify  an  existing  microscopic  imaging  system  for  acquiring  low 
magnification  (1  pixel=  5  pm)  images  of  entire  tissue  sections  and  for  tracing  in  3D 
the  ducts  in  the  tissue  specimen  from  a  series  of  images  of  adjacent  sections. 

1  Complete  the  existing  JAVA  based  software  for  interactive  marking  and  3D 
virtual  rendering  of  ducts  so  that  it  allows  any  branching  pattern.  (Months  1-6 ) 

2.  Develop  a  customized  VRML  viewer  to  allow  visualization  of  the  branching 
pattern  and  hyperlink  it  to  the  original  images  (Month  6-12 ) 

Task  2.  (Months  6-30)  Interface  the  existing  acquisition  and  registration  software  with 
the  JAVA  application  to  allow  revisiting  of  acquired  slides  for  inspection  and  high- 
resolution  acquisition  of  areas  of  interest.  (Months  6-12) 

1.  Integrate  the  JAVA  application  with  the  conventional  fluorescence  microscope 
( Months  6-12) 

2.  Integrate  the  JAVA  application  with  the  confocal  microscope  (Month  12-18) 

3.  Integrate  the  high-resolution  images  in  the  VRML  3D  representation  of  the 
mammary  gland  (Months  12-20) 

4.  Integrate  the  image  analysis  methods  for  nuclear  segmentation  with  the  JAVA 
application  (Months  18-24) 

5.  Integrate  the  results  of  the  segmentation  with  the  VRML  representation  of  the 
mammary  gland.  (Month  24-30) 

ACOMPLISHMENTS 


Under  the  combined  budgetary  efforts  of  this  and  grant  DAMD1 7-00-1 -0306,  we  have 
developed  a  client-server  application,  following  the  scheme  shown  in  Figure  1.  The 
server  is  a  C  application  that  runs  on  a  computer  (Dell  Inspiron,  running  Solaris  7  for 
Intel)  physically  attached  to  an  Axioplan  (Zeiss  Inc.,  Germany)  microscope.  The  server 
provides  access  to  the  drivers  that  control  the  moving  parts  of  the  microscope:  motorized 
scanning  stage,  excitation  filter  wheel  and  arc-lamp  light  blocking  shutter  (Ludl 
Electronic  Products  Ltd.,  Hawthorne,  NY),  as  well  as  to  the  controller  of  the  CCD 
Microimager  camera  (Xillix  Techonologies  Corp.,  Richmond,  British  Columbia,  Canada). 
Besides  giving  access  to  basic  operations,  such  as  acquiring  and  storing  images,  setting 
the  exposure  time  of  the  CCD  or  moving  the  stage  and  or  the  filter  wheel,  the  server  has 
been  programmed  to  offer  more  complex  functions.  Some  of  them  are  automatic 
focusing  of  the  microscope  or  acquisition  of  multiple-field  of  view  images.  To  do  this,  the 
server  receives  the  order  and  divides  it  into  a  set  of  simple  actions.  For  instance,  to 
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acquire  a  multiple  field-of-view  images,  the  server  takes  the  coordinates  of  the  vertices 
of  the  area  to  acquire  and  then  automatically  performs  the  sequence  of  stage 
movements  and  camera  operations  for  first  creating  a  focus  map  of  the  area  and  then 
acquiring  a  set  of  consecutive  images  that  covers  the  area.  The  output  is  a  mosaic-like 
image  of  the  area.  The  server  permits  multi-color  acquisition  in  fluorescence,  by 
performing  consecutive  acquisition  using  different  excitation  filters. 

The  client  (insofar  referred  as  R3D2)  is  connected  to  the  server  through  UNIX 
sockets,  which  are  the  standard  for  Internet-based  communications.  This  means  that  it 
can  send  requests  to  the  server  from  any  computer  internet-connected  to  the  server.  To 
obtain  the  maximum  benefit  from  this  fact,  R3D2  has  been  written  in  JAVA  (v.1.2),  so 
that  it  can  be  executed  on  different  computer  platforms. 

Figure  2  shows  the  R3D2’s  Graphical  User  Interface  (GUI).  The  interface  is  divided 
in  two  distinguishable  parts.  One  -rightmost  vertical  panel  in  Figure  2-  provides 
connections  to  the  server  and  allows  the  user  to  request  its  services  through  a  user- 
friendly  interface.  The  available  actions  are: 

•  Connect:  Establishes  a  new  connection  with  the  server  via  an  Internet 

socket.  All  the  following  options  assume  that  this  action  has  been 
successful. 

•  Set  Objective  lens:  Our  system  does  not  have  a  motorized  objective  tower, 

and  therefore  this  option  does  not  effectively  change  the  objective  lens 
positioned  in  the  light  path.  However,  the  user  must  update  the  lens  when 
manually  changing  the  lenses,  because  this  information  is  required  for 
some  of  the  remaining  options. 

•  Change  Filter:  Changes  the  excitation  filter.  This  is  needed  when  using 

fluorescence  microscopy. 

•  Exposure  Time:  Sets  the  exposure  time  for  the  CCD  camera  (in  ms.) 

•  Set  Origin:  This  option  moves  the  scanning  stage  of  the  microscope  to  its 

absolute  mechanical  origin  of  coordinates,  relative  to  which  all  future 
movements  and  measurements  are  done.  This  is  required  to  precisely 
locate  the  areas  of  the  slides  where  images  are  taken,  which  is  necessary 
for  accurate  revisiting  of  the  slide  and  correct  reconstruction  of  the  tissue. 

•  Acquire  image:  Takes  an  image  using  the  existing  settings.  R3D2  receives 
the  image  from  the  server  and  displays  it,  both  complete  -zoomed  out-  as 
well  as  partially  -in  their  original  resolution-.  Only  a  small  part  of  the  full- 
resolution  image  can  be  displayed  at  full  resolution.  The  area  can  be 
interactively  determined  by  moving  a  window  on  the  complete  version  of 
the  image  (Figure  3).  Images  can  be  saved  both  in  ICS  [Dean  90]  and 
JPEG  format.  When  images  are  saved  as  ICS,  all  the  acquisition 
parameters  (objective,  filter,  location  of  the  image  on  the  slide  are,  etc.) 
are  stored  in  the  ICS  header  file. 
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•  Autofocus:  Automatically  focuses  the  microscope  by  taking  a  series  of 

images  at  different  positions  in  the  Z  axis  (step  size=0.5  microns)  and 
determining  the  best  focused  image  of  the  series.  Blur  due  out  of  focus 
light  reduces  image  contrast,  which  can  be  detected  using  several 
functions.  Based  on  several  comparisons  described  in  the  existing 
literature  [Groen  85][Firestone  91][Santos  97],  we  selected  an 
autocorrelation-based  measurement  introduced  by  Vollath  [Vollath  87]. 

•  Open/Close  Shutter:  Closes  the  arc  lamp  light  blocking  shutter,  to  reduce 

photobleaching  due  to  unnecessary  light  exposure.  This  is  useful  when 
using  fluorescence  microscopy. 

•  Scan:  Acquires  multiple  field  of  view  images.  The  system  displays  a 

dialogue-panel  (Figure  4)  where  the  user  can  specify  the  filter(s)  to  be 
used  (in  fluorescence  microscpy),  exposure  time(s)  and  the  limits  of  the 
area  to  acquire.  The  limits  can  be  defined  by  its  coordinates  (when 
known)  or  manually,  by  moving  the  microscope  to  the  upper,  lower, 
rightmost  and  leftmost  points  of  the  area.  Figure  5  shows  an  example  of 
an  image  of  a  complete  tissue  section. 

•  Revisit  Point:  When  the  user  clicks  on  a  point  of  a  previously  imaged 

section,  the  server  moves  the  stage  to  that  location  on  the  slide  and  takes 
an  image  using  the  current  values  of  objective,  filter  and  exposure  time. 

•  Revisit  Area:  When  using  the  option,  the  user  is  prompted  to  draw  a 

rectangle  on  the  image  of  a  section.  Then  a  dialogue  panel  similar  to  the 
one  used  to  acquire  sections  asks  the  user  for  the  parameters  of  the 
acquisition(s).  Then  the  system  automatically  acquires  the  area(s).  This 
can  be  used  to  re-acquire  areas  using  a  different  magnification  and/or 
filter  (when  using  fluorescence).  This  feature  greatly  simplifies  some 
tasks,  such  as  determining  and  acquiring  areas  of  interest  (e.g.  sections 
of  mammary  ducts,  blood  vessels,  tumor  micro-invasions,  etc.)  on  whole 
tissue  sections.  Using  our  system  one  can,  for  instance,  acquire  a  whole- 
view  of  a  section  at  low  magnification  (x2.5,  xIO),  and  then  revisit  as 
many  areas  as  he/she  wants  for  acquisition  (e.g.  all  the  ducts  on  a 
section  of  mammary  gland)  at  higher  magnification  (x40,  x63,  xlOO).  The 
process  is  shown  in  Figure  6.  Multiple  areas  within  a  section  can  be 
background-acquired  by  the  system  without  any  human  interaction.  (See 
Task  2.1) 

Besides  providing  access  to  the  server  functions.  The  second  part  of  the  interface 
(two  left  vertical  panels  in  Figure  2)  expands  the  system  capabilities,  by  allowing  creating 
and  handling  sets  of  related  images,  which  we  call  Cases.  A  case  can  have  multiple- 
color,  multiple-magnification  related  images,  easily  accessible  and  meaningfully  stored. 

For  us,  a  case  is  a  sequence  of  low-magnification  images  of  complete  tissue 
sections  taken  from  a  tissue  block,  along  with  all  the  areas  re-visited  on  them  at  higher 
resolution  and/or  with  different  filters.  Finally  the  user  can  add  information  to  the  cases, 
by  marking  and/or  delineating  structures  of  interest  and  linking  them  within  and  between 
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consecutive  sections  (See  Task  1.1).  These  are  the  types  of  marks  and  some  actions 
that  can  be  performed  related  to  them: 

•  Text:  Textual  annotations  to  identify  structures  of  interest  for  future  study  or 

to  draw  the  attention  of  a  different  user. 

•  Duct:  Identifies  ductal  structures.  A  section  of  a  duct  perpendicular  to  the 

sectioning  plane  is  marked  with  a  circle  in  the  center  lumen.  Each  circle 
has  a  unique  number  that  identifies  it.  Its  coordinates  are  stored  in  a  data 
structure  that  contains  information  on  all  the  ducts  and  their 
interconnections. 

•  Connected  Ducts:  Connects  ductal  marks.  Within  the  same  section,  an 

elongated,  longitudinal  section  of  a  duct,  is  normally  marked  with  two 
Duct  marks  connected  by  a  line.  If  both  ends  are  located  on  different 
sections,  the  connection  is  displayed  by  changing  the  color  of  both  Duct 
marks  when  the  mouse  travels  over  any  one  of  them. 

•  Shapes:  Delineates  irregularly  shaped  ducts  or  other  structures  (e.g.  tumor 

masses,  lymph  nodes,  etc.). 

•  Groups:  Groups  series  or  Shapes  corresponding  to  consecutive  sections. 

The  Shapes  are  grouped  by  sequentially  clicking  on  them  while  the  option 
to  create  Groups  is  on.  To  show  grouping,  the  system  changes  the  color 
of  all  the  Shapes  of  the  group  every  time  the  mouse  moves  into  any  one 
of  them.  Groups  identify  connected  objects,  such  as  sections  of  a  tumor 
mass,  lymph  nodes,  etc. 

•  See  Area:  Displays  previously  acquired  images  of  Areas  of  interst,  when 

the  user  clicks  on  the  rectangle  that  locates  them  within  their 
corresponding  image  section. 

Before  reconstructing  a  Case  in  3D,  all  its  sections  must  be  registered.  For  that, 
we  calculate  the  Rigid-Body  Transform  that  gives  the  optimum  rotation  and  translation 
between  each  pair  of  sections.  The  Transform  is  calculated  from  six  pairs  of  points 
interactively  marked  by  the  user  on  each  pair  of  images  to  be  registered.  Once  the 
points  have  been  marked,  the  software  calculates  the  rotation  and  translation  (0,  tx,  ty) 
needed  to  minimize  the  sum  of  the  squared  distances  between  all  six  pairs  of 
corresponding  points.  The  results  are  stored  in  the  ICS  header  file  of  the  second  image. 
When  registering  a  Case,  all  sections  are  aligned  with  the  first  section  of  the  case,  either 
by  directly  registering  them  to  the  first  section  or  by  using  already  corrected  coordinates 
of  contiguous  sections. 

This  method  is  very  accurate  when  the  pairs  of  points  are  spread  all  along  the 
sections.  Reasons  for  small  errors  are  imprecise  mouse  interaction,  stretching  and/or 
compression  of  the  tissue,  and  the  fact  that  some  structures  used  to  select  pairs  of 
corresponding  points  might  not  be  perpendicular  to  the  sections.  Using  as  many  pairs  of 
points  as  possible  can  average  out  these  small  errors.  That  is  why,  instead  of  using  just 
four  pairs  of  registration  points  -which  is  the  minimum  required  for  this  method  to  work- 
we  use  six. 
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An  important  feature  of  R3D2  is  that  all  Case-handling  and  marking  functions  can  be 
used  in  parallel  to  the  functions  that  request  microscope  actions.  Therefore,  acquiring  a 
new  section  can  be  done  in  parallel  to  any  other  Case  related  function  (e.g.  registering 
already  acquired  sections  or  annotating  the  images).  This  is  possible  because  we  are 
using  Solaris  threads.  Threads  permit  executing  multiple  parallel  copies  of  a  program 
without  multiplying  resource  use.  Each  thread  shares  memory  and  other  resources  with 
other  threads.  R3D2  runs  on  a  main  thread  and  when  a  microscope-based  operation  is 
selected,  it  launches  a  new  thread  that  runs  on  the  same  memory  space  as  the  main 
one.  Thus,  the  client  can  accept  handling  and  marking  operations  on  the  main  thread 
while  a  microscope-based  operation  is  executed  at  the  new  one.  This  also  guarantees 
that,  in  the  case  of  a  microscope  failure  or  a  socket  error,  the  system  will  not  die 
abruptly,  as  only  the  thread  working  on  the  microscope  will  be  affected. 

The  most  innovative  feature  of  our  system  consists  on  its  ability  to  reconstruct  the 
tissue  structures  by  rendering  the  user  markings  in  3D  (See  Task  1.1).  For  that,  we  have 
used  Java3D  (v.  1.2.1.)  This  is  an  Application  Programming  Interface  (API)  for  3D  in 
Java.  After  asking  the  user  for  the  range  of  sections  to  render,  the  system  converts  the 
coordinates  of  all  the  markings  in  that  range  of  sections  from  two-dimensional  to  three- 
dimensional  values.  The  section  number  and  thickness,  along  with  the  distance  between 
sections,  determines  the  Z  -depth-  coordinate.  Then  a  3D  scene  is  built  using  several 
geometric  shapes  to  represent  the  different  markings:  spheres  for  sections  of  ducts 
perpendicular  to  the  section,  lines  for  connections  ,  polygons  for  shapes,  ...  Once  the 
Case  is  rendered  (using  another  thread)  it  is  displayed  in  a  new  window  where  the 
mouse  can  be  used  to  rotate  the  scene,  to  zoom  in  or  out  or  to  translate  the  scene  in  the 
X  and/or  Y  directions. 

The  3D  window  includes  a  tool  bar  with  options  to  select  elements.  Selecting 
elements  one  can  obtain  information  about  them,  load  the  image(s)  that  contain  that 
element  or  move  the  microscope  to  its  location  on  the  slide  for  re-imaging.  That  way,  by 
just  clicking  on  elements  of  the  3D  scene,  the  user  can  both  display  and/or  revisit  areas 
of  interest  which  could  have  been  otherwise  missed  using  a  conventional  2D  analysis 
(See  Task  1.2).  The  window  also  has  buttons  to  hide/show  connections,  shapes,  areas 
(acquired  at  higher  magnification)  and  the  three-dimensional  coordinate  axis.  There  is 
also  a  button  to  reset  the  scene  to  the  default  view  and  combo  boxes  to  the  change  the 
scale  in  any  of  the  three  dimensions. 

We  will  now  illustrate  the  use  of  our  system  by  showing  the  steps  followed  to  image 
and  reconstruct  two  different  tissue  blocks.  These  blocks  were  used  to  test  the  system. 
Both  blocks  have  equal  tissue  origin  -mouse  mammary  gland-  and  correspond  to  two 
different  modalities  of  optical  microscopy.  The  first  block  is  an  H&E  stained,  paraffin 
embedded  gland,  the  second  is  a  fluorescently  counterstained,  frozed  gland. 

Tissue  source 

Female  BALB/c  mice  were  obtained  from  Simonson  (Gilroy,  CA)  and  housed  4 
per  cage  with  chow  and  water  ad  libitum  in  a  temperature  and  light  controlled  facility. 
Carbon  dioxide  inhalation  was  used  to  kill  the  animals  in  accordance  with  Association  for 
Assessment  and  Accreditation  of  Laboratory  Animal  Care  guidelines  and  institutional 
review  and  approval.  The  3rd  and  4th  inguinal  glands  were  removed  for  histology  and 
whole  mounts  at  4,  6,  8  &  10  weeks. 
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Tissue  preparation 

One  gland  (MP1,10  weeks  old)  was  paraffin-embedded,  H&E  stained.  A  second 
gland  (MF1,  8  weeks  old)  was  frozen  and  immerse  in  Evan’s  Blue,  which  stains  both 
nuclear  and  cytoplasmatic  areas.  Evan's  Blue  fluorescent  emission  is  bright  red,  similar 
to  Texas  Red.  Both  glands  were  sectioned  at  4  pm,  and  placed  on  glass  slides  for 
microscopy  and  imaging. 

Imaging  of  tissue  sections 

Full  tissue  sections  were  imaged  in  bright  field  (MP1),  or  fluorescence  (MF1), 
using  R3D2’s  Scan  option  (See  Figure  7).  Sections  were  imaged  at  10X,  with  a  Fluar 
(0.5  na)  objective  lens  (Zeiss,  Wetzlar,  Germany).  In  order  to  match  the  emission  of 
Evan’s  Blue,  a  filter  set  for  imaging  Texas  Red  was  used.  To  optimize  memory  use  while 
keeping  enough  resolution  for  Histology,  images  were  reduced  by  a  factor  of  4  in  both  X 
and  Y  directions,  which  gave  as  and  effective  2.5X  magnification,  i.e.  a  sixteen  fold 
reduction  in  image  size.  After  image  compression,  the  average  image  was  lOMbytes. 
This  reduction  was  necessary  to  speed  up  image  transmission  from  the  server  and 
display  on  the  client  GUI. 

Creation  of  Cases 

Two  cases  were  created  which  included  all  sections  from  each  of  the  imaged 
glands.  (See  Figure  8).  Cases  were  created  empty,  and  images  of  the  sections  added  as 
they  were  being  acquired.  MP1  had  24  sections,  from  approximately  half  of  the 
mammary  gland,  while  MF1  had  92  sections,  which  corresponded  to  a  fully  sectioned 
complete  mammary  gland. 

Tissue  Annotation 

All  sections  were  manually  annotated.  We  marked  the  centers  of  the  ducts  by 
placing  a  circular  mark  (R3D2’s  Duct  tool)  in  the  lumen  -when  the  duct  was 
perpendicular  to  the  image  plane-  or  with  a  line  -when  the  duct  was  sectioned 
longitudinally-.  Then  we  connected  the  markings  within  and  between  consecutive 
sections  using  the  Connect  Ducts  option.  We  also  delineated  the  lymph  nodes  using 
R3D2’s  Shapes  option  and  grouped  their  consecutive  sections  using  the  Group  options. 

3D  Reconstruction 

Figure  9  shows  the  rendering  of  the  annotations  done  on  the  cases.  The 
reconstruction  is  fully  interactive  in  that  by  clicking  on  the  markings  one  can:  i)  retrieve 
information  (location,  image  statistics  of  the  surrounding  area);  ii)  display  the  area  of  the 
original  image  that  contains  that  annotation  or  iii)  automatically  move  the  microscope  to 
that  location,  if  the  appropriate  slide  is  place  on  the  stage. 


PROBLEMS 


The  main  problems  faced  so  far  are: 

•  The  size  of  the  images,  when  acquiring  entire  tissue  sections  even  at  low 
magnification  (lOx),  is  quite  large  (130  Mb.  or  more).  This  exhausts  the  computer 
memory  when  more  than  one  full-section  is  displayed,  even  using  a  computer 
with  512  Mb  of  RAM  memory,  the  reason  being  the  high  memory  consumption  of 
the  Operating  System  and  the  JAVA  Virtual  Machine,  which  leave  limited 
resources  for  the  client  requirements.  In  order  to  make  the  system  work  at  a 
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reasonably  speed  without  compromising  its  use,  we  introduced  a  compression 
factor  which  can  be  set  by  the  user  before  acquiring  a  section.  The  user  can 
choose  between  acquiring  the  image  at  full  resolution  (10X),  half  resolution 
(effective  5x)  or  at  one  forth  of  the  original  resolution  (effective  2.5X).  Using  the 
highest  compression  factor  (14)  is  recommended  when  imaging  tissue  sections 
larger  that  1  cm2.  Smaller  tissue  sections  can  be  acquired  using  half  or  full 
resolution.  Consistency  requires  maintaining  equal  resolution  for  all  sections  of  a 
case. 

•  Scanning  whole  sections  is  an  unsupervised  operation,  which  uses  parameters 
previously  defined  by  the  user.  The  exposure  time  is  one  of  these  parameters. 
That  value  is  set  before  starting  scanning  and  does  not  change  until  the  scanning 
is  done.  In  fluorescence,  this  is  a  problem,  since  the  effect  of  the  preparation 
protocols  (e.g.  protein  digestion,  nuclear  permeabilization  in  FISH  or 
Immunohistochemistry,  etc.)  varies  from  cell  type  to  cell  type.  Thus  the 
concentration  of  fluorescent  material  that  reaches  its  target,  may  it  be  the 
chromatin,  a  gene  or  a  protein,  varies  between  different  parts  of  the  tissue. 
Therefore,  the  exposure  time  that  is  appropriate  for  one  part  of  the  tissue  may 
not  be  appropriate  for  another.  That  affects  the  quality  of  the  images,  since  some 
areas  tend  to  be  either  overexposed  or  underexposed.  At  the  moment  we 
manually  selected  an  intermediate  value  that  allows  us  to  image  dark  areas 
without  overexposing  bright  areas.  This  works  in  almost  all  cases,  but  renders 
images  with  varying  contrast.  To  avoid  that,  we  are  developing  an  algorithm 
which,  when  activated  by  the  user,  will  adjust  the  exposure  time  to  the 
fluorescence  intensity. 

•  An  important  problem  is  that,  due  to  tissue  complexity,  image  annotation  of  real 
tissue  specimens  is  very  time  consuming.  This  is  true,  even  when  reconstructing 
fully  sectioned  mammary  glands  of  young  animals  (6  weeks).  In  that  specific 
case,  the  average  number  of  ducts  per  section  is  100,  having  all  of  them  to  be 
marked  and  connected  to  their  corresponding  ducts  on  the  two  contiguous 
sections.  If  the  gland  was  fully  sectioned  at  e.g.  4  microns,  each  case  has  an 
average  of  100  sections,  leaving  us  with  10000  markings  and  40000  connections 
per  case,  which  is  absolutely  impracticable.  However,  most  of  our  studies 
scheduled  for  the  two  coming  years  don’t  require  fully  sectioning  of  glands,  since 
a  correct  planning  the  sectioning  will  reduce  the  number  of  sections  to 
approximately  20.  There  is  also  the  immediate  option  of  increasing  the  distance 
between  sections,  using  only  one  every  N  sections  for  the  reconstruction.  N  has 
to  be  low  enough  as  to  ensure  that  it  is  still  possible  to  determine  ductal 
continuity  between  adjacent  sections.  We  are  now  in  the  process  of  determining 
the  optimum  value.  Finally,  we  are  working  on  algorithms  to  automate  the 
detection  of  ducts.  The  options  are  to  fully  automatic  detection  and  reconstruction 
the  ducts,  followed  by  interactive  correction,  or  manual  marking  of  an 
intermediate  slide  and  then  automatic  ‘growth’  of  the  structures  from  that  slide 
towards  the  contiguous  sections. 

•  A  final  problem,  related  to  the  previous  one,  is  the  redundancy  of  information  in 
the  3D  reconstruction.  Marking  the  same  structure  on  consecutive  sections 
sometimes  leads  to  3D  objects  with  too  much  level  of  detail.  This  can  complicate 
the  interpretation  of  the  results.  We  are  working  on  the  way  to  ‘prune’  the  3D 
reconstruction  of  the  markings  and  create  and  skeletonized  version  of  it. 
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Task  3.  (Months  6-36)  Use  #1  &  #2  to  perform  3-D  reconstruction  maps  of  the 
distribution  of  ER  and  PR  in  the  ductal  tree  and  determine  whether  ER  colocalizes 
with  PR  in  mouse  mammary  epithelium  during  critical  phases  of  mammary 
development. 

1.  Select  the  animals  (20)  to  study,  taken  at  different  stages  of  development. 
Extract  fix  and  section  the  tissue  as  explained  in  the  Methods  Section  ( Months 
6-12.4  specimens:  Months  12-24.  8  specimens .  Months  24-36,  8  specimens ) 

2.  Reconstruct  the  mammary  gland  using  the  software  developed  according  Task  1. 
(Months  12-24,  5  specimens;  Months  24-36,  15  sections) 

3.  Revisit  areas  of  interest  for  high  resolution  imaging  and  detection/colocalization 
of  Progesterone  and  Estrogen  receptors)  (Months  24-36,  20  sections)  Integrate 
the  information  (Months  30-36) 

For  the  reporting  period,  the  only  scheduled  sub-task  was  selecting  the  glands  that 
will  be  analyzed  and  reconstructed  in  the  following  two  years.  Given  that  the  sample 
preparation  time  is  relatively  short  (2-3  days)  and  has  to  be  done  immediatly  before  the 
sections  are  imaged,  we  have  delayed  selecting  the  animals  and  preparing  the  samples 
until  right  before  they  are  analyzed. 
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KEY  RESEARCH  ACCOMPLISHMENTS 


The  first  year  of  the  grant  was  used  to  develop  the  microscopy  and  computer 
infrastructure  that  will  be  used  in  the  following  two  years.  In  summary: 

•  We  have  developed  a  3D  computerized  microscopy  platform  that  allows  image 
acquisition  of  entire  tissue  sections  at  low  resolution,  and  then  provides 
interactive  tools  for  marking  structures  of  interest  (in  our  case  mammary  ducts, 
tumor  masses,  etc.).  The  system  can  reconstruct  the  annotations  in  3D  to 
provide  a  better  understanding  of  tissue  organization,  which  can  be  used  to  plan 
high-resolution  analysis  of  areas  of  interest. 

•  The  3D  reconstruction  is  linked  to  the  microscope,  to  allow  revisiting  and  re¬ 
acquisition  of  areas  of  interest  at  high  magnification  and  multiple  fluorescent 
channels  (when  using  fluorescence  microscopy). 

•  To  test  the  system,  we  have  successfully  imaged  and  reconstructed  two 
mammary  glands,  stained  with  different  methods  (H&E  and  fluorescence).  By 
doing  it  we  have  achieved  an  understanding  of  the  system  that  will  help  us  to 
design  the  research  goals  proposed  for  the  coming  two  years. 
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REPORTABLE  OUTCOMES 


Manuscripts: 

•  “Recent  advances  in  quantitative  digital  image  analysis  and  applications  in  Breast 
Cancer*’.  Ortiz  de  Solorzano  C.,  Callahan  D.E.,  Parvin  B.,  Costes  S.,  Barcellos-Hoff, 
M.H.  Review  paper  submitted  to  Microscopy  Research  and  Technique. 

•  “Application  of  three-dimensional  image  analysis  to  the  study  of  Breast  Cancer1’. 

Ortiz  de  Solorzano  C.  Proceedings  of  the  XX  Congress  of  the  Spanish  Society  of 
Pathology,  pp.  253-257  (In  Spanish),  Pamplona,  Spain  July  1-5,  2001, 2001. 

•  “A  system  for  computer-based  reconstruction  of  3-dimensional  structures  from  serial 
tissue  sections:  an  application  to  the  study  of  normal  and  neoplastic  mammary  gland 
biology”.  Fernandez-Gonzalez,  R.,  Jones  A.,  Garcia-Rodriguez  E.,  Knowles  D., 
Sudar  D.  Ortiz  de  Solorzano,  C.  Proceedings  of  Microscopy  and  Microanalysis’2001 . 
Microscopy  and  Microanalysis  7(suppl.  2),  pp.  964-5,  2001. 

Presentations: 

•  Application  of  three-dimensional  image  analysis  to  the  study  of  Breast  Cancer.  XX 
Congress  of  the  Spanish  Society  of  Pathology.  Pamplona,  Spain  July  1-5,  2001 , 

2001 .  Invited  talk 

•  A  system  for  computer-based  reconstruction  of  3-dimensional  structures  from  serial 
tissue  sections:  an  application  to  the  study  of  normal  and  neoplastic  mammary  gland 
biology.  Microscopy  and  Microanalysis’01,  Long  Beach,  CA  August  5th-9th,  2001 . 
Platform  presentation. 


Informatics: 

•  As  described  in  the  Body  of  the  report  and  in  the  Reportable  Outcomes  sections,  we 
have  developed  a  three-dimensional  computerized  3D  microscopy  platform  (R3D2). 


Funding  applied: 

•  Segmentation  of  Mammary  Gland  Ductal  Structure  Using  Geometric  Methods.  P.I.’s 
Malladi  R.  and  Ortiz  de  Solorzano  C.  Submitted  to  the  LBNL  Laboratory  Directed 
Research  and  Development  Program  (LDRD)  2002,  in  the  Strategic-Computational 
Sub-Program. 
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Employment  or  Research: 

•  Upon  reception  of  the  Award,  I  was  offered,  and  accepted,  a  Scientist  Position  at  the 
Life  Sciences  Division,  Lawrence  Berkeley  National  Laboratory  of  the  University  of 
California. 

•  The  outstanding  work  of  my  Research  Assistant  (Rodrigo  Fernandez  Gonzalez)  on 
the  development  of  the  microscopy  and  computing  infrastructure  has  been 
instrumental  in  his  acceptance  to  the  joint  UCSF-UC  Berkeley  Ph.D.  program  in 
Biomedical  Engineering.  He  will  continue  working  part-time  on  this  project  as 
Graduate  Student  Research  Assistant. 
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CONCLUSIONS 


In  summary,  the  goals  for  the  reporting  year  have  been  fully  accomplished.  We 
have  developed  a  three-dimensional  computerized  microscopy  platform  that  allows 
imaging  and  3D-reconstruction  of  tissue  blocks  to  combine  morphological  (Histo- 
Pathology)  and  molecular  (Immunohistochemistry)  information.  Our  platform  directs  and 
simplifies  image  acquisition  at  different  resolution  levels  and  rationalizes  image  storage 
and  retrieval. 

Applying  our  system  to  the  test  cases  described  in  this  report  has  helped  us 
understanding  some  practical  limitations  of  the  described  methodology  (e.g.  high 
memory  and  storage  needs  for  the  data  sets  or  high  interactive  burden  when  manually 
annotating  tissue  structure).  Practical  solutions  have  been  proposed  and  implemented, 
and  therefore  the  existing  system  is  appropriate  for  the  analysis  planned  for  the  following 
two  years  -studying  the  whole-gland  distribution  and  interplay  of  Estrogen  and 
Progesterone  receptors  in  normal  and  neoplastic  mammary  gland  development-. 
However,  we  will  devote  some  limited  resources  to  exploring  further  automation  of  the 
two  most  time-consuming  tasks:  image  acquisition  and  tissue  annotation.  Imaging  of 
whole  sections  can  be  sped  up  by  using  an  automatic  slide  feeder  attached  to  the 
microscope,  so  that  all  sections  can  be  acquired  without  requiring  manual  interaction. 
Tissue  annotation  interaction  can  be  greatly  reduced  by  using  existing  image  analysis 
algorithms  for  automatically  detecting  tissue  structures  (Ducts,  lymph  nodes)  and 
connecting  them  between  sections.  By  using  them,  interaction  will  be  then  limited  to 
annotating  one  section  of  the  case  and  correcting  the  results  of  the  automatic  detection. 

The  relevance  of  what  we  have  achieved  is  significant,  since  our  system  can  and 
will  eventually  be  applied  to  a  variety  of  studies,  both  molecular  and  clinical,  which  will 
greatly  profit  from  the  additional  -third-  dimension.  Several  Pathologists  have  expressed 
their  interest  in  our  system,  arguing  that  it  will  improve  the  way  they  look  at  the  samples 
and  the  way  they  handle  and  store  ‘visual’  information.  Full  use  of  this  system  in  a 
clinical  setting  requires  further  automation  in  the  areas  above  mentioned,  to  provide 
additional  information  without  a  substantial  increase  of  diagnostic  time.  Similarly, 
several  colleagues  in  the  Breast  Cancer  Research  field  have  expressed  their  interest  in 
our  system,  which  would  allow  them  to  do  whole  sample  or  whole  gland  studies  similar 
to  the  one  proposed  in  this  grant. 
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FIGURES 


Java  Client  Server  Microscope 
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Figure  1 .  Description  of  the  system 

The  application  uses  client-server  architecture.  The  server  runs  on  a 
computer  physically  attached  to  the  microscope,  and  provides  access  to  the 
microscope  functions.  The  client  is  a  JAVA  application,  which  can  run  on  any 
computer  (no  particular  OS  required)  connected  to  the  server  through  the 
Internet  Protocol  (IP).  The  communication  between  client  and  server  uses 
UNIX  sockets.  The  client  provides  user-friendly  access  to  all  the  microscope 
functions  offered  by  the  server,  and  allows  handling  of  sets  of  related  images 
(Cases)  for  storage,  annotation  and  3D  reconstruction  of  structures  of 
interest. 
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Figure  2.  R3D2’s  Graphical  User  Interface. 

The  client  application  (R3D2)  is  a  JAVA  application.  The  GUI  is  divided  in  two  main 
parts.  The  two  leftmost  panels  are  for  displaying  consecutive  sections  of  a  Case.  It 
also  provides  the  user  with  tools  for  registering  the  sections,  annotating  the  images 
and  connecting  structures  between  sections  (e.g.  mammary  ducts,  tumor 
volumes),  and  reconstructing  the  annotated  images  in  3D.  The  rightmost  panel 
gives  access  to  the  microscope  related  functions  offered  by  the  Server. 
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Figure  3.  Acquisition  of  a  single  image 

To  acquire  a  single  image,  the  user  clicks  on  the  “Acquire”  button,  after  choosing 
the  appropriate  microscope  settings:  objective  lens,  excitation  filter  (in 
fluorescence)  and  exposure  time  of  the  CCD  camera.  The  “Autofocus”  option 
automatically  finds  the  correct  focus  plane  of  the  microscope  before  acquiring  an 
image.  The  top  panel  contains  the  entire  image  (reduced  to  fit  in  the  window),  while 
the  lower  panel  contains  a  zoomed  version  of  the  part  of  the  original  image 
selected  by  the  rectangle  on  the  top  window.  The  rectangle  can  be  manually 
moved  to  look  at  different  parts  of  the  image  at  its  original  -full-  resolution. 
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Figure  4.  Panel  for  acquisition  of  whole-sections 

To  acquire  multiple  field-of-view  images,  the  user  is  fills  out  this  form,  which 
requests  the  name  of  the  file  where  the  image  will  be  stored  (if  not  only  displayed), 
the  number  of  scans  -in  fluorescence,  and  which  filter  and  exposure  time  are  to  be 
used  in  each  scan,  and  the  coordinates  of  the  limits  of  the  area.  The  coordinates 
can  be  entered  in  the  boxes,  if  known,  or  otherwise  interactively  defined  at  the 
microscope. 
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Figure  5.  Whole  section  acquisition 

This  image  shows  the  result  of  scanning  a  complete  tissue  section  from  a  tissue 
block  of  a  patient  with  Ductal  Carcinoma  In  Situ  of  the  Breast  (DCIS),  stained  for 
Histology  with  Hematoxylin  and  Eosin.  As  fo  the  individual  images,  the  upper  panel 
contains  the  entire  section  while  the  lower  contains  a  zoomed  sub-area  of  it. 
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3D  Reconstruction 
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Figure  6.  Revisiting  of  Areas  of  Interest 


Top.  R3D2  showing  a  case  of  a  tissue  block  from  a  patient  with  Ductal  Carcinoma 
In  Situ  of  the  Breast.  The  block  was  sectioned  and  stained  with  H&E  (odd  sections) 
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and  DAPI  -counterstain-  and  FISH  to  the  erb-b2  gene.  The  interface  shows  two 
sections.  The  left  section  (the  fluorescent  one)  shows  several  light  (yellow) 
rectangles  which  correspond  to  areas  which  were  revisited  and  reacquired  at  a 
higher  resolution.  By  clicking  on  a  rectangle,  the  user  can  display  those  areas  in 
new  panels,  which  show,  in  this  case,  a  counterstained  area  (Bottom,  left)  and  the 
image  channel  corresponding  to  the  emission  of  the  fluorochrome  attached  to  the 
probe  to  the  erb-b2  gene  hybridized  on  the  section  (Bottom,  right). 


Figure  7.  Tissue  acquisition. 


The  image  shows  two  sections  from  the  cases  used  to  test  the  systems.  Left. 
Tissue  section  of  an  H&E  stained  mouse  mammary  gland.  Right.  Tissue  section  from  a 
mouse  mammary  gland  stained  with  Evan’s  Blue,  which  stains  both  nuclei  and 
cytoplasm.  Microscope  focusing  and  image  acquisition  is  completely  automated,  as 
described  in  the  text. 
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Figure  8.  Tissue  annotation. 

Two  consecutive  H&E  (a)  and  Evan’s  Blue  (b)  stained  sections  of  the  two  mouse 
mammary  glands  using  to  test  the  system.  Ducts  are  connected  within  and  between 
sections.  Lymph  node  countours  are  drawn  and  also  connected  between  consecutive 
sections.  Connections  between  sections  are  shown  by  changing  line  color  of  all 
connected  components  every  time  one  of  them  is  visited  (selected  or  just  traveled  over 
by  the  mouse  pointer). 
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Figure  9.  Tissue  Reconstruction. 

3D  reconstruction  of  the  manually  annotated  cases  used  to  test  our  system:  (a)  H&E 
stained,  (b)  Case  stained  with  Evan’s  Blue.  Lymph  node  volumes  have  been  surface- 
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rendered  to  show  their  three-dimensional  shapes.  Ducts  are  identified  by  spheres  and 
connected  by  lines  within  and  between  sections.  The  3D  reconstruction  is  fully 
interactive.  It  can  be  rotated,  zoomed,  and  all  the  elements  are  active,  in  that  b  y  clicking 
on  them  one  can  retrieve  information  about  the  element,  display  the  original  image  that 
contains  the  element  or  automatically  move  the  microscope  to  the  selected  point. 


